{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 指标计算(整体对应evalution_segmentation.py)\n",
    "\n",
    "1. 相关指标简介\n",
    "2. 混淆矩阵形成过程\n",
    "3. 指标计算过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c0p4mNaXsAnH"
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import numpy as np\n",
    "import six\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9qlFzQoBlGVS"
   },
   "source": [
    "### Documentation of funcs:\n",
    "Evaluate metrics used in Semantic Segmentation.\n",
    "\n",
    "This function calculates Intersection over Union (IoU), Pixel Accuracy\n",
    "and Class Accuracy for the task of semantic segmentation.\n",
    "\n",
    "The definition of metrics calculated by this function is as follows,\n",
    "where $N_{ij}$ is the number of pixels that are labeled as class $i$ by the ground truth and class $j$ by the prediction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hsEcR6eCi3wW"
   },
   "source": [
    "$\\text{IoU of the i-th class} =  \\frac{N_{ii}}{\\sum_{j=1}^k N_{ij} + \\sum_{j=1}^k N_{ji} - N_{ii}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6MSXyaSpjdFD"
   },
   "source": [
    "$\\text{mIoU} = \\frac{1}{k} \n",
    "        \\sum_{i=1}^k \n",
    "        \\frac{N_{ii}}{\\sum_{j=1}^k N_{ij} + \\sum_{j=1}^k N_{ji} - N_{ii}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8TNmxQErju1W"
   },
   "source": [
    "$\\text{Pixel Accuracy} =  \n",
    "        \\frac \n",
    "        {\\sum_{i=1}^k N_{ii}} \n",
    "        {\\sum_{i=1}^k \\sum_{j=1}^k N_{ij}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QZRYxL0dkHkm"
   },
   "source": [
    "$\\text{Class Accuracy} = \n",
    "        \\frac{N_{ii}}{\\sum_{j=1}^k N_{ij}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZD_e2jvEkRim"
   },
   "source": [
    "$\\text{Mean Class Accuracy} = \\frac{1}{k} \n",
    "        \\sum_{i=1}^k \n",
    "        \\frac{N_{ii}}{\\sum_{j=1}^k N_{ij}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gW4y9JtykiaJ"
   },
   "source": [
    "The number of classes $n\\_class$ is $max(pred\\_labels, gt\\_labels) + 1$, which is the maximum class id of the inputs added by one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TTc8mnexmG26"
   },
   "source": [
    "The more detailed description of the above metrics can be found in a review on semantic segmentation Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, Jose Garcia-Rodriguez.  [A Review on Deep Learning Techniques Applied to Semantic Segmentation](https://arxiv.org/abs/1704.06857>). arXiv 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 655
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1146,
     "status": "ok",
     "timestamp": 1586170497676,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "N-ERgFsoFpA1",
    "outputId": "69af69a3-e72c-4c6e-ecb0-bedca382b6e7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 1, 11,  9],\n",
      "         [ 6,  4, 11]]])\n",
      "========================================\n",
      "tensor([[[[-3.6364, -2.5271, -2.6061],\n",
      "          [-3.6993, -3.7676, -3.7860]],\n",
      "\n",
      "         [[-2.8848, -2.2626, -1.8794],\n",
      "          [-1.3549, -1.9828, -1.0390]],\n",
      "\n",
      "         [[-2.4099, -2.7261, -2.3141],\n",
      "          [-2.5346, -3.4321, -2.5286]],\n",
      "\n",
      "         [[-3.8668, -2.6031, -3.1582],\n",
      "          [-2.8502, -4.0885, -3.3225]],\n",
      "\n",
      "         [[-3.7301, -2.7621, -3.6439],\n",
      "          [-2.5944, -2.1272, -3.3256]],\n",
      "\n",
      "         [[-4.0690, -2.2244, -1.8543],\n",
      "          [-2.1628, -4.8737, -5.3196]],\n",
      "\n",
      "         [[-1.9503, -4.2843, -1.8384],\n",
      "          [-3.6383, -4.4557, -1.6903]],\n",
      "\n",
      "         [[-1.1397, -4.2204, -2.9541],\n",
      "          [-1.8096, -1.2834, -4.5804]],\n",
      "\n",
      "         [[-1.6517, -1.5704, -3.0781],\n",
      "          [-2.3460, -4.1107, -2.9618]],\n",
      "\n",
      "         [[-3.4865, -1.7748, -2.1138],\n",
      "          [-3.2565, -1.7787, -4.2973]],\n",
      "\n",
      "         [[-4.3939, -2.5988, -2.8379],\n",
      "          [-3.1677, -2.0229, -2.5436]],\n",
      "\n",
      "         [[-2.6719, -3.6926, -4.3535],\n",
      "          [-3.7267, -2.8635, -2.0541]]]])\n"
     ]
    }
   ],
   "source": [
    "# fake label data\n",
    "gt = np.random.rand(1, 2, 3)*12\n",
    "gt = gt.astype(np.int64)\n",
    "gt = torch.from_numpy(gt)\n",
    "\n",
    "x = torch.randn(1, 12, 2, 3)  # 假设每个像素点可以对应12个分类\n",
    "out = F.log_softmax(x, dim=1)\n",
    "\n",
    "print(gt)\n",
    "print('='*40)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1146,
     "status": "ok",
     "timestamp": 1586170502630,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "HnOwqk6IFuW0",
    "outputId": "ef088915-204d-40bb-8c5e-e0e487e96ffd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[7, 8, 6],\n",
      "       [1, 7, 1]])]\n",
      "[array([[ 1, 11,  9],\n",
      "       [ 6,  4, 11]])]\n"
     ]
    }
   ],
   "source": [
    "pred_labels = out.max(dim=1)[1].data.cpu().numpy()  # ndarray，预测值对应成真实分类，映射成整数对应具体分类\n",
    "pred_labels = [i for i in pred_labels]  # list\n",
    "print(pred_labels)  # \n",
    "gt_labels = gt.data.cpu().numpy()\n",
    "gt_labels = [i for i in gt_labels]\n",
    "print(gt_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1109,
     "status": "ok",
     "timestamp": 1586170276227,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "VNBoxzdT-IcF",
    "outputId": "bfb63377-4d31-4906-86eb-48adc686e5e0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 2\n",
      "(2, 3) (2, 3)\n"
     ]
    }
   ],
   "source": [
    "for pred_label, gt_label in six.moves.zip(pred_labels, gt_labels): # six.moves python2和3通用的申明\n",
    "    print(pred_label.ndim, gt_label.ndim)\n",
    "    print(pred_label.shape, gt_label.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 691,
     "status": "ok",
     "timestamp": 1586170278581,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "jtIWl95gGaoM",
    "outputId": "095e4439-04ef-47f2-da94-3b251c17c961"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8 3 7 9 6 8]\n",
      "[10 11  7  3  4 10]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 55,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_label, gt_label = pred_labels[0], gt_labels[0]\n",
    "pred_label = pred_label.flatten()\n",
    "gt_label = gt_label.flatten()\n",
    "print(pred_label)\n",
    "print(gt_label)\n",
    "np.max((pred_label, gt_label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1124,
     "status": "ok",
     "timestamp": 1586170520151,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "3nFISDecFPgN",
    "outputId": "b478c0cd-ad65-4608-ee3e-b025e3a44f25"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True,  True,  True,  True])"
      ]
     },
     "execution_count": 59,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算bincount之前需要转为布尔值，不能有负值，有则剔除\n",
    "mask = gt_label >= 0 \n",
    "mask     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1012,
     "status": "ok",
     "timestamp": 1586173331890,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "DnfAB1ifTDza",
    "outputId": "f2ccf844-03c3-4b04-dfbd-e59a2f42b874"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 11,  7,  3,  4, 10])"
      ]
     },
     "execution_count": 64,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gt_label[mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1192,
     "status": "ok",
     "timestamp": 1586173363014,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "iKlSh2_tTJ74",
    "outputId": "84f36913-cac7-47ed-a318-bac9d9386270"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([120, 132,  84,  36,  48, 120])"
      ]
     },
     "execution_count": 65,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "12*gt_label[mask] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1024,
     "status": "ok",
     "timestamp": 1586173390600,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "oLofm7_9TO71",
    "outputId": "555f8208-83ef-4a7a-dca4-354a145f7c14"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([128, 135,  91,  45,  54, 128])"
      ]
     },
     "execution_count": 66,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "12*gt_label[mask] + pred_label[mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8Fxo0b5mcVkR"
   },
   "source": [
    "### [numpy.bincount](https://docs.scipy.org/doc/numpy/reference/generated/numpy.bincount.html)\n",
    "　　Each bin gives the number of occurrences of its index value in x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rfHDfToUb8KE"
   },
   "outputs": [],
   "source": [
    "# >>> np.bincount(np.arange(5))\n",
    "# array([1, 1, 1, 1, 1])\n",
    "# >>> np.bincount(np.array([0, 1, 1, 3, 2, 1, 7]))\n",
    "# array([1, 3, 1, 1, 0, 0, 0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 101
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1185,
     "status": "ok",
     "timestamp": 1586174282888,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "X-Awz80mV7Cc",
    "outputId": "a939264f-edce-4c39-8fd2-f984228135f0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(144,)\n",
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "confusion_onedim = np.zeros((144,), dtype=np.int64) #初始化12类的平方\n",
    "confusion_onedim += np.bincount(\n",
    "        12 * gt_label[mask].astype(int) + pred_label[mask],   # 类似12进制了\n",
    "        minlength=12 ** 2)\n",
    "print(confusion_onedim.shape)\n",
    "print(confusion_onedim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 218
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1261,
     "status": "ok",
     "timestamp": 1586177045789,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "IoE1utR-Si47",
    "outputId": "cbbab81c-9cff-433a-9202-dbee7deab205"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0],\n",
       "       [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 78,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion = np.zeros((12, 12), dtype=np.int64)\n",
    "confusion += np.bincount(\n",
    "        12 * gt_label[mask].astype(int) + pred_label[mask],   # 类似12进制了\n",
    "        minlength=12 ** 2).reshape((12, 12))\n",
    "confusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1481,
     "status": "ok",
     "timestamp": 1586177071619,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "rkz94S2hhL1V",
    "outputId": "b8293823-dee8-4fb8-9102-a29271d5dc6d"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATQAAAD8CAYAAAD5TVjyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAXM0lEQVR4nO3dfbBdVX3G8e/jTRBBC8FUCknEdJpR\nqVbRDGBxKopitErs1HGSVo0OTpyOKNpOO2gdo+gftHasOqJ4B1OkpYk2Yk2dKGRAh7YWmvBSIYnU\nGJXcGA0xiC8IIcnTP/a+9JzDvfece8++52Xf5zOz556z9j57/aLjz7X2Wmsv2SYiog6e0O8AIiKq\nkoQWEbWRhBYRtZGEFhG1kYQWEbWRhBYRtZGEFhGzRtISSd+QtFPSDkmXTnCNJH1S0m5J35b0goZz\nayR9tzzWtK0v89AiYrZIOg04zfYdkp4C3A68zvbOhmteDbwTeDVwDvAJ2+dIOgXYDiwHXP72hbYf\nmKy+tNAiYtbY3m/7jvLzL4BdwKKWy1YC17pwK3BymQhfCWy1fahMYluBFVPVN6/yf8EUpBMMJ/ey\nyog55mfYD6mbO/yO5Ic6vHY/7AAebigatT060bWSngGcBdzWcmoRsLfh+1hZNln5pHqa0Ipktra3\nVUbMKRPmkml5CHh7h9d+EB62vbzddZKeDHwJeLftn3cR3pTS5YyIJqJo6XRydHQ/aT5FMrvO9vUT\nXLIPWNLwfXFZNln5pJLQIqLJE4AndXi0I0nA54Bdtj82yWWbgTeXo53nAg/a3g/cAFwoaYGkBcCF\nZdmketzljIhBJ2B+dbc7D3gTcLeku8qy9wFPB7B9FbCFYoRzN0WP963luUOSPgxsK393ue1DU1WW\nhBYRTca7nFWw/R/lLae6xsA7Jjm3HljfaX1ddTklrZB0bzkh7rJu7hURg2G8hdbJMWhmnIgljQBX\nAq+gGE7dJmlz44S5iBg+VbbQeq2buM8GdtveAyBpI8UEuSS0iCFW8TO0nuomoU006e2c1oskreWx\nyWcndVFdRPTC+CjnMJr1lmU5a3gUQDo9C0cjBtxcbaFNe9JbRAyHufgMbRuwTNJSikS2CviTSqKK\niL6Zky0020ckXUIxc3cEWG97R2WRRURfzNVRTmxvoZjlGxE1kUGBiKiNOdnljIh6mrNdzoion7TQ\nIqI20kKLiNpICy0iakNklDMiakLA/E4zw5HZjGT6ktAiookE85LQIqIOJJg/0u8oZiYJLSKaTKuF\n1vZeWg+8Bjhg+zkTnP9L4E/Lr/OAZwO/We4n8APgF8BR4Egn2+UloUVEEwnmP7Gy210DfAq4dqKT\ntj8KfLSoV68F3tOyEcpLbR/stLIktIhoVuFENNu3lDumd2I1sKGb+rIvZ0Q0q3qn4U6qlE4AVlBs\nSDzOwI2Sbi/ffN1WWmgR8XidZ4aFkrY3fB8t31I9Xa8F/rOlu/li2/skPQ3YKuk7tm+Z6iZJaBHR\nTBRvOOzMwU4e1ndgFS3dTdv7yr8HJH2ZYmOmKRNaupwR0azHXU5JJwEvAb7SUHaipKeMfwYuBO5p\nd6+00CKimYCKRjklbQDOp+iajgHrKJeK2r6qvOyPgBtt/6rhp6cCX5YERZ76Z9tfb1dfElpENKt2\nlHN1B9dcQzG9o7FsD/C86daXhBYRzYb4/UFDGnZEzKosfYqIWkgLLSJqIwktImqjwlHOXktCi4hm\naaFFRG0koUVEbUxv6dNASUKLiGZpoUVEbQzxoMCMF6dLWiLpG5J2Stoh6dIqA4uIPunD+9Cq0k1I\nR4C/sH1HuSr+dklbbe+sKLaI6Ie52OW0vR/YX37+haRdwCIgCS1i2M21hNaofGf4WcBtE5xbC5Sv\nzz2piuoiYjbN5VFOSU+meA/4u23/vPV8+Tre0eLa091tfRExy+ZilxNA0nyKZHad7eurCSki+mqI\nRzlnnNBUvEryc8Au2x+rLqSI6KshbqF1s6fAecCbgJdJuqs8Xl1RXBHRL3Nx2obt/6D4p0dEnczR\nFlpE1NVIh0cbktZLOiBpwh2bJJ0v6cGGXt4HGs6tkHSvpN2SLusk7CHNwxExa6ptoV0DfAq4dopr\n/t32a5pCkEaAK4FXAGPANkmb203cT0KLOen9PlzZvT6i4yq710CocJTT9i3lPNXpOhvYXe7+hKSN\nwEraTNxPlzMimk1vUGChpO0Nx9oZ1PgiSf8j6WuSfrcsWwTsbbhmrCybUlpoEdFsel3Og7aXd1Hb\nHcAZtn9ZzpL4V2DZTG+WFlpENBtf+lTBoEA7tn9u+5fl5y3AfEkLgX3AkoZLF5dlU0oLLSKa9XDa\nhqTfAn5i25LOpmhk/RT4GbBM0lKKRLYK+JN290tCi4hmAo6v6FbSBuB8imdtY8A6YD6A7auA1wN/\nJukI8GtglW0DRyRdAtxA0RZcb3tHu/qS0CKiWYVv27C9us35T1FM65jo3BZgy3TqS0KLiGZDvFJg\nSMOOiFk1pJlhSMOOiFkzl1/wGBE1ky5nRNTGXHzBY0TUVFpoEVEbSWgRURtJaBFRKxnljIhaSAst\nImojo5wRURtpoUVdVfWq6kF7TfWgxTNQktAiojaS0CKiTpxRzoioAz8BDlf0gsdeS0KLiCYWHBnp\ndLuRY7May3QloUVEE0scnddpaqhuf9MqdL3rk6QRSXdK+moVAUVE/x0dGenoaEfSekkHJN0zyfk/\nlfRtSXdL+pak5zWc+0FZfpek7Z3EXUUL7VJgF/AbFdwrIvrMiKPVrX26hmLPgGsnOf994CW2H5D0\nKmAUOKfh/EttH+y0sq5aaJIWA38IXN3NfSJicBhxhJGOjrb3sm8BDk1x/lu2Hyi/3kqx/+aMddtC\n+zjwV8BTJrug3Bq+3B7+pC6ri4jZZsThztc+LWzpDo7aHp1h1RcDX2sKBW6UZOCzndx3xglN0muA\nA7Zvl3T+ZNeVQYwWvzndM60vInpjml3Og7aXd1unpJdSJLQXNxS/2PY+SU8Dtkr6Ttnim1Q3Xc7z\ngIsk/QDYCLxM0j91cb+IGBBHGenoqIKk36N4bLXS9k/Hy23vK/8eAL4MnN3uXjNOaLbfa3ux7WdQ\nbNN+s+03zvR+ETEYqnyG1o6kpwPXA2+y/b8N5SdKesr4Z+BCYMKR0kaZhxYRTYouZzWpQdIG4HyK\nZ21jwDpgPoDtq4APAE8FPi0J4EjZhT0V+HJZNg/4Z9tfb1dfJVHb/ibwzSruFRH9VQwKVPM2Etur\n25x/G/C2Ccr3AM97/C+mlhZaRDQxVNKd7IcktIhoUV2Xs9eGM+qImDUVrxToqSS0mFLe7Do3JaFF\nRC2khRYRtWHEI0O67VMSWkQ0SQstImojCS0iaiXz0CKiFqpc+tRrwxl1RMyadDkjojaKUc7hnH+Y\nhBYRTdLljIhaSZczImohz9AiojaS0CKiNrL0KSJqY5hbaF1tNBwR9VTVrk+S1ks6IGnCDU5U+KSk\n3ZK+LekFDefWSPpueazpJO4ktIhoUvGuT9cAK6Y4/ypgWXmsBT4DIOkUig1VzqHYvm6dpAXtKktC\ni4gm4/PQOjna3qvYGPjQFJesBK514VbgZEmnAa8Etto+ZPsBYCtTJ0Ygz9AiYgLTeIa2UNL2hu+j\ntkenUdUiYG/D97GybLLyKSWhRUSTaW5jd7DcR3MgpMsZEU16uXM6sA9Y0vB9cVk2WfmUktAiokmV\nz9A6sBl4cznaeS7woO39wA3AhZIWlIMBF5ZlU0qXMyIep6p5aJI2AOdTPGsboxi5nA9g+ypgC/Bq\nYDfwEPDW8twhSR8GtpW3utz2VIMLQBJaRLSocmKt7dVtzht4xyTn1gPrp1NfV11OSSdL2iTpO5J2\nSXpRN/eLiP7r8TO0SnXbQvsE8HXbr5d0HHBCBTFFRB8Vo5xzbC2npJOAPwDeAmD7MHC4mrAiol/m\n6lrOpcD9wD9IulPS1ZJObL1I0lpJ24vJdw91UV1E9EpVazl7rZuENg94AfAZ22cBvwIua73I9qjt\n5cXku/RIIwbdMD9D6yahjQFjtm8rv2+iSHARMcR6PA+tUjOOyPaPJe2V9Ezb9wIXADurCy0i+mGa\nS58GSrcp9p3AdeUI5x7KSXERMbzGu5zDqKuEZvsuYGAWpkZENQaxO9mJ4Yw6ImbNME/bSEKLiCZJ\naBFRK3PyGVpEp97vahaRfETDOfo2TI7xhLm39Cki6itdzoiohTxDi4jaMHmGFhG1ocxDi4h6SJcz\nImrDiEcqXMspaQXFy2BHgKttX9Fy/u+Bl5ZfTwCeZvvk8txR4O7y3H22L5qqriS0iGjiCruckkaA\nK4FXULyhZ5ukzbYfe5GF7fc0XP9O4KyGW/za9vM7rS/b2EXE41T4gsezgd2295Rvtd4IrJzi+tXA\nhpnGnYQWEU3Gn6F1mNAWjr+RujzWttxuEbC34ftYWfY4ks6geBP2zQ3Fx5f3vVXS69rFni5nRDQx\n4uixjgcFDhZvo67EKmCT7aMNZWfY3ifpt4GbJd1t+3uT3SAJLSKa+Jh45OHKlj7tA5Y0fF9clk1k\nFS17dNreV/7dI+mbFM/XJk1o6XJGRBNbHD0y0tHRgW3AMklLyxfBrgI2t14k6VnAAuC/GsoWSHpi\n+XkhcB5t3oqdFlpENDOdJqv2t7KPSLoEuIFi2sZ62zskXQ5stz2e3FYBG8ud1Mc9G/ispGMUja8r\nGkdHJ5KEFhFNbHHk0eom1treAmxpKftAy/cPTvC7bwHPnU5dSWgR0UIcOzqcqWE4o46I2WOgoi5n\nryWhRUSzY4KHhzM1DGfUMXTyptkhc6TfAcxMElpENCteiDaUktAiolkSWkTUhoFH+x3EzCShRUQz\nA4/0O4iZSUKLiGZD3OXsai2npPdI2iHpHkkbJB1fVWAR0SfjCa2TY8DMOKFJWgS8C1hu+zkU67RW\nVRVYRPTJECe0bruc84AnSXqU4l3gP+o+pIjoq7nY5SzfU/R3wH3AfuBB2ze2Xidp7fjbLOGhmUca\nEb0zpC20brqcCyjeDb4UOB04UdIbW6+zPWp7efFWyxNmHmlE9MYx4OEOjwHTzaDAy4Hv277f9qPA\n9cDvVxNWRPTNHH2Gdh9wrqQTgF8DFwDbK4kqIvpniJ+hzTih2b5N0ibgDop//p3AaFWBRUSfzMWE\nBmB7HbCuolgiYlAMaULLJikR0aziZ2iSVki6V9JuSZdNcP4tku6XdFd5vK3h3BpJ3y2PNe3qytKn\niGh2jOKpeAUkjQBXAq+g2GR4m6TNE2x28gXbl7T89hSKHuByijR7e/nbByarLy20iGhm4GiHR3tn\nA7tt77F9GNhIMd2rE68Etto+VCaxrcCKqX6QhBYRj9d5l3Ph+MT58ljbcqdFwN6G72NlWas/lvRt\nSZskjW9M3OlvH5MuZ0Q0m94o58Fi0nxX/g3YYPsRSW8HPg+8bCY3SgstIppVOyiwD1jS8H1xWfb/\n1dk/tT3+BrargRd2+ttWSWgR0azapU/bgGWSlko6juKNPJsbL5B0WsPXi4Bd5ecbgAslLSiXWl5Y\nlk0qXc6IeLyK5qHZPiLpEopENAKst71D0uXAdtubgXdJuqis9RDwlvK3hyR9mCIpAlxu+9BU9cl2\nNZF3QDrd0PrMMCKqM4r9I3VzBz1tuXl9h6sYP6PbK3iGVpm00CKiWTZJiYjaGJ+HNoSS0CKi2Vxd\nnB4RNWQqW/rUa0loEdEsXc6IqI10OSOiNpLQIqI2Mm0jImolz9AiohbG13IOoSS0iGiWLmdE1Eam\nbURErWSUM2L2reNDldznQ9l9cXKZthERtZFBgYiojbTQIqJWktAiohaGeNpG201SJK2XdEDSPQ1l\np0jaWm7PvrXcwCAi6qDajYaRtELSvZJ2S7psgvN/LmlnuS/nTZLOaDh3VNJd5bG59betOtn16Roe\nv1vxZcBNtpcBN5XfI6IOKtzGTtIIcCXwKuBMYLWkM1suuxNYbvv3gE3A3zac+7Xt55fHRe3qa5vQ\nbN9CsRNLo5UUm4FS/n1du/tExJA4RvGCx06O9s4GdtveY/swsJEifzzG9jdsP1R+vZVi/80Zmem+\nnKfa3l9+/jFw6kwDiIgBVF2XcxGwt+H7WFk2mYuBrzV8P17Sdkm3SmrbcOp6UMC2JU26F56ktTy2\nd91J3VYXEb3Q+e6WCyU17nk3ant0JlVKeiOwHHhJQ/EZtvdJ+m3gZkl32/7eZPeYaUL7iaTTbO8v\ndz0+MNmF5T9utAj49N5tAhoRvXCwzb6c+4AlDd8Xl2VNJL0c+GvgJbYfGS+3va/8u0fSN4GzgEkT\n2ky7nJuBNeXnNcBXZnifiKi3bcAySUslHQesosgfj5F0FvBZ4CLbBxrKF0h6Yvl5IXAesHOqytq2\n0CRtAM6naFqOAeuAK4AvSroY+CHwho7/eRExZ9g+IukS4AZgBFhve4eky4HttjcDHwWeDPyLJID7\nyhHNZwOflXSMovF1he3uEprt1ZOcuqDTf1REDJPxYc5q2N4CbGkp+0DD55dP8rtvAc+dTl1ZKRAR\nLYZ3qUASWkS0GN7V6UloEdEiLbSIqI0ktDnt/T5c2b0+ouMqu1cd5U2zvWCqHBTopSS0iGiRZ2gR\nURvpckZEbaSFFhG1kRZaRNRGWmgRURvVLn3qpSS0iGiRLmdE1Eq6nBFRC2mhRURtJKFFRG1klDMi\naiOjnBFRG+lyRkRtDG+Xc6a7PkVEbY230Do52pO0QtK9knZLumyC80+U9IXy/G2SntFw7r1l+b2S\nXtmuriS0iGgx3kLr5JiapBHgSuBVwJnAaklntlx2MfCA7d8B/h74m/K3Z1Jse/e7wArg0+X9JpWE\nFhEtxgcFOjnaOhvYbXuP7cPARmBlyzUrgc+XnzcBF6jYz24lsNH2I7a/D+wu7zepHj9D238QPvTD\nNhctBA72IpoOtY3nI+pRJIVB+88HBi+muRzPGd3fYv8N8MGFHV58vKTtDd9HbY82fF8E7G34Pgac\n03KPx64p9/F8EHhqWX5ry28XTRVMTxOa7d9sd42k7W22lu+pxNPeoMWUeLpje0W/Y5ipdDkjYjbt\nA5Y0fF9clk14jaR5wEnATzv8bZMktIiYTduAZZKWSjqO4iH/5pZrNgNrys+vB2627bJ8VTkKuhRY\nBvz3VJUN4jy00faX9FTiaW/QYko8A6J8JnYJcAMwAqy3vUPS5cB225uBzwH/KGk3cIgi6VFe90Vg\nJ8WQ6jtsH52qPhWJMCJi+KXLGRG1kYQWEbUxMAmt3fKIPsSzRNI3JO2UtEPSpf2OCYqZ15LulPTV\nAYjlZEmbJH1H0i5JL+pzPO8p/7u6R9IGScf3IYb1kg5Iuqeh7BRJWyV9t/y7oNdxzRUDkdA6XB7R\na0eAv7B9JnAu8I4BiAngUmBXv4MofQL4uu1nAc+jj3FJWgS8C1hu+zkUD6BX9SGUayiW6TS6DLjJ\n9jLgpvJ7zIKBSGh0tjyip2zvt31H+fkXFP9jnXKW8myTtBj4Q+DqfsZRxnIS8AcUI1TYPmz7Z/2N\ninnAk8q5TCcAP+p1ALZvoRipa9S4tOfzwOt6GtQcMigJbaLlEX1NHo3K1f9nAbf1NxI+DvwVxWK7\nflsK3A/8Q9kFvlrSif0KxvY+4O+A+4D9wIO2b+xXPC1Otb2//Pxj4NR+BlNng5LQBpakJwNfAt5t\n++d9jOM1wAHbt/crhhbzgBcAn7F9FvAr+tiVKp9LraRItKcDJ0p6Y7/imUw5YTRzpWbJoCS0aS9x\n6AVJ8ymS2XW2r+9zOOcBF0n6AUWX/GWS/qmP8YwBY7bHW62bKBJcv7wc+L7t+20/ClwP/H4f42n0\nE0mnAZR/D/Q5ntoalITWyfKInipfX/I5YJftj/UzFgDb77W92PYzKP7zudl231ogtn8M7JX0zLLo\nAooZ3f1yH3CupBPK/+4uYHAGTxqX9qwBvtLHWGptIJY+TbY8os9hnQe8Cbhb0l1l2ftsb+ljTIPm\nncB15f8J7QHe2q9AbN8maRNwB8UI9Z30YcmRpA3A+cBCSWPAOuAK4IuSLgZ+CLyh13HNFVn6FBG1\nMShdzoiIriWhRURtJKFFRG0koUVEbSShRURtJKFFRG0koUVEbfwfLvWdjc2mUKEAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm = confusion\n",
    "plt.figure()\n",
    "plt.grid(False)\n",
    "plt.imshow(cm, cmap='jet')\n",
    "plt.colorbar()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7SRS7K3CdinS"
   },
   "source": [
    "### [next()](https://www.programiz.com/python-programming/methods/built-in/next)\n",
    "The syntax of next() is:\n",
    "\n",
    "`next(iterator, default)`\n",
    "\n",
    "next() Parameters:\n",
    "\n",
    "* iterator - next() retrieves next item from the iterator\n",
    "\n",
    "* default (optional) - this value is returned if the iterator is exhausted (there is no next item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rEb4clCAoAb5"
   },
   "outputs": [],
   "source": [
    "def calc_semantic_segmentation_confusion(pred_labels, gt_labels):\n",
    "    \"\"\"Collect a confusion matrix. 计算 混淆矩阵\n",
    "    The number of classes `n_class` is `max(pred_labels, gt_labels) + 1`, which is\n",
    "    the maximum class id of the inputs added by one.\n",
    "    Args:\n",
    "        pred_labels(iterable of numpy.ndarray): A collection of predicted\n",
    "            labels. The shape of a label array\n",
    "            is `(H, W)`. `H` and `W`\n",
    "            are height and width of the label.\n",
    "        gt_labels(iterable of numpy.ndarray): A collection of ground\n",
    "            truth labels. The shape of a ground truth label array is\n",
    "            `(H, W)`, and its corresponding prediction label should\n",
    "            have the same shape.\n",
    "            A pixel with value `-1` will be ignored during evaluation.\n",
    "    Returns:\n",
    "        numpy.ndarray:\n",
    "        A confusion matrix. Its shape is `(n_class, n_class)`.\n",
    "        The `(i, j)` th element corresponds to the number of pixels\n",
    "        that are labeled as class `i` by the ground truth and\n",
    "        class `j` by the prediction.\n",
    "    \"\"\"\n",
    "    pred_labels = iter(pred_labels)\n",
    "\n",
    "    gt_labels = iter(gt_labels)\n",
    "\n",
    "    n_class = 12\n",
    "    # 定义一个数值容器 shape(12,12)\n",
    "    confusion = np.zeros((n_class, n_class), dtype=np.int64)\n",
    "    \n",
    "    for pred_label, gt_label in six.moves.zip(pred_labels, gt_labels):  # six.moves.zip in python2\n",
    "        if pred_label.ndim != 2 or gt_label.ndim != 2:\n",
    "            raise ValueError('ndim of labels should be two.')\n",
    "        if pred_label.shape != gt_label.shape:\n",
    "            raise ValueError(\n",
    "                'Shape of ground truth and prediction should be same.')\n",
    "        pred_label = pred_label.flatten()\n",
    "        gt_label = gt_label.flatten()\n",
    "\n",
    "        # Dynamically expand the confusion matrix if necessary.\n",
    "        lb_max = np.max((pred_label, gt_label))\n",
    "        # print(lb_max)\n",
    "        if lb_max >= n_class:\n",
    "            expanded_confusion = np.zeros(\n",
    "                (lb_max + 1, lb_max + 1), dtype=np.int64)\n",
    "            expanded_confusion[0:n_class, 0:n_class] = confusion\n",
    "\n",
    "            n_class = lb_max + 1\n",
    "            confusion = expanded_confusion  # 原来的confusion矩阵就没有了，被expanded_confusion替换了\n",
    "\n",
    "        # Count statistics from valid pixels. \n",
    "        mask = gt_label >= 0\n",
    "        confusion += np.bincount(\n",
    "            n_class * gt_label[mask].astype(int) + pred_label[mask],  # 这样处理axis=0 代表gt axis=1 代表pred……\n",
    "            minlength=n_class ** 2) \\                  # ……即 横表示gt ; 列表示pred \n",
    "            .reshape((n_class, n_class))  # 抓住一个点，混淆矩阵中，对角线上的点是分类正确的\n",
    "\n",
    "    for iter_ in (pred_labels, gt_labels):\n",
    "        # This code assumes any iterator does not contain None as its items.\n",
    "        if next(iter_, None) is not None:\n",
    "            raise ValueError('Length of input iterables need to be same')\n",
    "\n",
    "    # confusion = np.delete(confusion, 11, axis=0)\n",
    "    # confusion = np.delete(confusion, 11, axis=1)\n",
    "    return confusion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rbfXIQZYoYVx"
   },
   "outputs": [],
   "source": [
    "def calc_semantic_segmentation_iou(confusion):\n",
    "    \"\"\"Calculate Intersection over Union with a given confusion matrix.\n",
    "    Args:\n",
    "        confusion (numpy.ndarray): A confusion matrix. Its shape is\n",
    "            `(n_class, n_class)`.\n",
    "            The `(i, j)`th element corresponds to the number of pixels\n",
    "            that are labeled as class `i` by the ground truth and\n",
    "            class `j` by the prediction.\n",
    "    Returns:\n",
    "        numpy.ndarray:\n",
    "        An array of IoUs for the `n_class` classes. Its shape is `(n_class,)`.\n",
    "    \"\"\"\n",
    "    # iou_denominator 并集  np.diag(confusion) 交集\n",
    "    iou_denominator = (\n",
    "        confusion.sum(axis=1) + confusion.sum(axis=0) - np.diag(confusion)) \n",
    "    iou = np.diag(confusion) / iou_denominator\n",
    "    return iou[:-1]  # 去掉最后一个类别，因为最后一个类别为 unlabelled\n",
    "    # return iou\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ji-XSm08o8mo"
   },
   "source": [
    "The definition of Intersection over Union (IoU) is as follows,\n",
    "where $N_{ij}$ is the number of pixels\n",
    "that are labeled as class $i$ by the ground truth and\n",
    "class $j$ by the prediction.\n",
    "\n",
    "$\\text{IoU of the i-th class} =  \\frac{N_{ii}}{\\sum_{j=1}^k N_{ij} + \\sum_{j=1}^k N_{ji} - N_{ii}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qMH3XZ3EeMIS"
   },
   "outputs": [],
   "source": [
    "def eval_semantic_segmentation(pred_labels, gt_labels):\n",
    "    \"\"\"Evaluate metrics used in Semantic Segmentation\n",
    "    Args:\n",
    "        pred_labels (iterable of numpy.ndarray): A collection of predicted\n",
    "            labels. The shape of a label array\n",
    "            is (H, W). H and W are height and width of the label.\n",
    "            For example, this is a list of labels [label_0, label_1, ...], \n",
    "            where label_i.shape = (H_i, W_i).\n",
    "        gt_labels (iterable of numpy.ndarray): A collection of ground\n",
    "            truth labels. The shape of a ground truth label array is\n",
    "            (H, W), and its corresponding prediction label should\n",
    "            have the same shape.\n",
    "            A pixel with value `-1` will be ignored during evaluation.\n",
    "    Returns:\n",
    "        dict:\n",
    "\n",
    "        The keys-value types and the description of the values are listed\n",
    "        below.\n",
    "        * iou(numpy.ndarray): An array of IoUs for the \n",
    "            `n_class` classes. Its shape is `(n_class,)`.\n",
    "        * miou(float): The average of IoUs over classes.\n",
    "        * pixel_accuracy(float): The computed pixel accuracy.\n",
    "        * class_accuracy(numpy.ndarray): An array of class accuracies \n",
    "            for the `n_class` classes. \n",
    "            Its shape is `(n_class,)`.\n",
    "        * mean_class_accuracy(float): The average of class accuracies.\n",
    "\n",
    "        Evaluation code is based on\n",
    "        https://github.com/shelhamer/fcn.berkeleyvision.org/blob/master/\n",
    "        score.py #L37\n",
    "\n",
    "    \"\"\"\n",
    "    confusion = calc_semantic_segmentation_confusion(pred_labels, gt_labels)\n",
    "    iou = calc_semantic_segmentation_iou(confusion)\n",
    "    pixel_accuracy = np.diag(confusion).sum() / confusion.sum()\n",
    "    class_accuracy = np.diag(confusion) / (np.sum(confusion, axis=1) + 1e-10)\n",
    "\n",
    "    return {'iou': iou, 'miou': np.nanmean(iou),\n",
    "        'pixel_accuracy': pixel_accuracy,\n",
    "        'class_accuracy': class_accuracy,\n",
    "        'mean_class_accuracy': np.nanmean(class_accuracy[:-1])}\n",
    "            # 'mean_class_accuracy': np.nanmean(class_accuracy)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1008,
     "status": "ok",
     "timestamp": 1586168091969,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "c2AfxY5n3Cpb",
    "outputId": "1322c67c-b0e2-4d12-fb41-53d7e979f6ac"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:18: RuntimeWarning: invalid value encountered in true_divide\n"
     ]
    }
   ],
   "source": [
    "eval_metrix = eval_semantic_segmentation(pred_labels, gt_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4QMjoy494kA0"
   },
   "source": [
    "**RuntimeWarning: invalid value encountered in true_divide**\n",
    "\n",
    "　　Maybe your code is trying to \"divide by zero\" or \"divide by NaN\". If you are aware of that and don't want it to bother you, then you can try:\n",
    "\n",
    "    import numpy as np\n",
    "    np.seterr(divide='ignore', invalid='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1001,
     "status": "ok",
     "timestamp": 1586168095566,
     "user": {
      "displayName": "Rongfan Liao",
      "photoUrl": "",
      "userId": "07803922812103577726"
     },
     "user_tz": -480
    },
    "id": "pAE3e3jG3dpb",
    "outputId": "91c4529f-9f13-48b5-a953-801c23b942d0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4666666666666667\n"
     ]
    }
   ],
   "source": [
    "print(eval_metrix['miou'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyNexNitaEb1X7o2nsd+7NpB",
   "name": "evaluate.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
